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Preface 



The Department of Homeland Security (DHS) Office of Inspector General (OIG) was established 
by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector 
General Act of 1978. This is one of a series of audit, inspection, and special reports prepared as 
part of our oversight responsibilities to promote economy, efficiency, and effectiveness within 
the department. 

This report addresses the strengths and weaknesses of DHS' process to determine, formulate, and 
address lessons learned and corrective needs identified during the Top Officials full-scale 
exercise series. It is based on interviews with employees and officials of relevant agencies and 
institutions, direct observations, quantitative analysis, and a review of applicable documents. 

The recommendations herein have been developed to the best knowledge available to our office, 
and have been discussed in draft with those responsible for implementation. We trust this report 
will result in more effective, efficient, and economical operations. We express our appreciation 
to all who contributed to the preparation of this report. 




Richard L. Skinner 
Inspector General 
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Executive Summary 

The Top Officials exercises test how key government officials respond to 
simulated terrorist attacks. The exercises are intended to strengthen the 
Nation's capacity to prepare for, prevent, respond to, and recover from 
large-scale terrorist acts. Our review examined the Department of 
Homeland Security's process to determine, formulate, and address lessons 
learned and corrective needs identified during Top Officials exercises. 

Since the first Top Officials exercises in 2000, neither a process for 
tracking weaknesses and how those weaknesses were resolved, nor a 
method for identifying and analyzing trends in corrective actions or 
significant lessons learned has been established. As a result, federal, state, 
local, and territorial agencies were unclear regarding the implementation 
of suggested improvements following preparedness exercises. 

In 2007, the Federal Emergency Management Agency instituted a federal 
interagency corrective action program to address these problems. This 
program established an interagency process for corrective action 
implementation and validation after Top Officials exercises. However, the 
corrective action process has not been fully implemented. The Federal 
Emergency Management Agency also established a National Exercise 
Program and standardized the terrorism exercise process. However, the 
Department of Homeland Security has not secured adequate participation 
and support from other federal departments and agencies; state, territorial, 
and local entities; or the private sector in planning, implementing, and 
evaluating exercises or in the corrective action processes. In addition, 
after-action reports, best practices, and lessons learned from Top Officials 
exercises have not been disseminated to a broad national audience. 

We are recommending that the Department of Homeland Security seek 
assistance from high-level agency and interagency committees, and amend 
National Exercise Program guidance, as needed, to (1) fully implement the 
corrective action program; (2) increase the level of participation by top 
officials in all phases of the exercise; and (3) disseminate after-action 
reports, best practices, and lessons learned to a broad national audience. 
Federal Emergency Management Agency officials generally agreed with 
our findings and will provide a detailed corrective action plan for 
implementation in their 90-day letter. 
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Background 



History of the TOPOFF Exercise Series 

In 1998, Congress provided funding for a national terrorism preparedness exercise 
to assess the nation's crisis and consequence management capability by applying 
plans, policies, procedures, systems, and facilities through federal, state, and local 
responses to terrorist threats and acts. Recognizing that top departmental and 
agency officials rarely participated fully in such exercises, Congress required "the 
participation of all key personnel who would participate in the consequence 
management of such an actual terrorist event." 1 In May 2000, the Department of 
Justice led the first in a series of national preparedness exercises called Top 
Officials (TOPOFF) 2000. 

TOPOFF 2000 simulated a pneumonic plague outbreak in Denver, Colorado, and 
a mustard gas attack in Portsmouth, New Hampshire. More than 6,500 federal, 
state, and local personnel, including top officials, responded to these simulated 
attacks. Important objectives of the exercise were to examine the interfaces 
between participating agencies and officials to identify seams, gaps, and 
redundancies that affect decision-making, and to identify subsequent actions to 
resolve consequence management issues. 

To build on TOPOFF 2000, Congress mandated the formation of the TOPOFF 
exercise series and the development of a continuous exercise cycle using 
seminars, tabletop exercises, and command post exercises that culminate in a 
biennial, national, full-scale TOPOFF exercise. Each exercise series would 
involve a 2-year cycle of seminars, planning events, and exercises, and would 
culminate in a full-scale assessment of the Nation's capacity to prevent, prepare 
for, respond to, and recover from large-scale terrorist attacks. 

In May 2003, the Department of Justice and the Federal Emergency Management 
Agency (FEMA) jointly sponsored TOPOFF 2, the second in the series of 
exercises, using the scenarios of a radiological dispersal device in Seattle, 
Washington, and pneumonic plague in Chicago, Illinois. TOPOFF 2 was 
designed to identify vulnerabilities in the domestic incident management 
capability by exercising against a series of integrated terrorist threats and acts. 
The exercise engaged 8,500 responders and top officials from the United States 
and Canada — the first international partner to participate. 

Because TOPOFF 2 was in its final planning stages when DHS was established, 
TOPOFF 3, the third exercise of the series, was the first exercise in which DHS 
maintained full responsibility for planning, conducting, and after-action reporting. 
Conducted in April 2005, TOPOFF 3 was designed to identify vulnerabilities in 
the Nation's domestic incident management capability. This exercise simulated 



1 Senate Report 105-235, July 2, 1998. 
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pneumonic plague attacks in Union and Middlesex counties, New Jersey, and a 
mustard gas release in New London, Connecticut. TOPOFF 3 included more than 
10,000 participants, with international exercise play in the United Kingdom and 
Canada as part of a partnership intended to strengthen security, communication, 
and information sharing among the three nations. 

During TOPOFF 3, federal departments and agencies had their first opportunity to 
use the National Response Plan (NRP). The NRP was designed to link national- 
level, hazard-specific contingency plans that could be implemented independently 
during localized incidents or concurrently during incidents of national significance. 2 
The foundation for the NRP was the National Incident Management System 
(NIMS), which provides a consistent national doctrinal framework for incident 
management at all jurisdictional levels, regardless of the cause, size, or complexity 
of the incident. 

During 2005, DHS developed a Universal Task List and Target Capabilities List 
to continue to define homeland security tasks, provide a framework to assess 
preparedness, and identify required resources. The Universal Task List describes 
incident management tasks to be performed, and provides a standardized 
reference for all levels of government and the private sector. The Target 
Capabilities List contains capabilities that various levels of government need to 
develop and maintain to prevent, respond to, and recover from a terrorist attack or 
major disaster. The Universal Task List and Target Capabilities List were still in 
the developmental stages during the TOPOFF 3 planning process and were 
subsequently integrated into the TOPOFF 4 exercise design, planning, evaluation, 
and improvement planning considerations. 

Homeland Security Exercise and Evaluation Program Policy and 
Guidance 

The purpose of the Homeland Security Exercise and Evaluation Program 
(HSEEP) is to provide common exercise policy and program guidance that 
constitutes a national standard for exercises. 3 Prior to 2007, HSEEP did not 
fully address exercise planning, evaluation, after-action implementation 
planning, or the corrective action process. In response to NRP and NIMS 
goals to standardize preparedness exercise design, development, and 
evaluation, DHS updated and expanded HSEEP guidelines for intra-agency 
exercises in Volume III, Exercise Evaluation and Improvement Planning, 
dated February 2007. Specifically, the updated 2007 HSEEP guidelines 
required that the TOPOFF 4 exercise include: 



2 The NRP was in effect for the TOPOFF 3 and TOPOFF 4 exercises but was replaced by the National Response 
Framework in January 2008. 

3 HSEEP policy and guidance is contained in five volumes: Volume I: Overview and Exercise Program 
Management; Volume II Exercise Planning and Conduct; Volume III: Exercise Evaluation and Improvement 
Planning; Volume IV: Sample Exercise Documents and Formats and Volume V: Prevention Exercises (Draft). 
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• Capabilities-based objectives in exercise design; 

• Data collected and analyzed according to HSEEP guidelines; 

• Exercise controller/evaluator training and briefings at exercise 
venues; 

• An after-action conference to discuss policy observations 
identified during the exercise; 

• A final after-action report (AAR) that includes an Improvement 
Plan; 

• Adherence to HSEEP security guidance regarding AAR and 
Improvement Plan distribution; and 

• Tracking and implementing of corrective actions through the 
Corrective Action Program (CAP). 

Following Homeland Security Council (HSC) guidance and HSEEP 
guidelines to plan exercises according to capabilities-based objectives, the 
October 2007 TOPOFF 4 exercise employed a radiological dispersal 
device scenario that involved coordinated terrorist attacks in Portland, 
Oregon; Phoenix, Arizona; and the U.S territory of Guam. 

Formation of the National Exercise Program 

In the December 2003, Homeland Security Presidential Directive (HSPD)-8 
National Preparedness, the President directed the Secretary of DHS "in 
coordination with other appropriate Federal departments and agencies, [to] 
establish a national program and a multi-year planning system to conduct 
homeland security preparedness-related exercises that reinforce training 
standards, provide for evaluation readiness, and support the national 
preparedness goal." Under HSPD-8, federal agencies were directed to 
participate in a collaborative, interagency process to design preparedness 
exercises on a consensus basis, and create a master exercise calendar. 

The HSC, created in October 2001 by HSPD-1: Organization and 
Operation of the HSC, is located in the Executive Office of the President. 
The HSC reports on matters related to homeland security and combating 
terrorism, and consists of cabinet secretaries and White House senior 
officials whose responsibility includes a principal interest in homeland 
security policymaking. The HSC maintains a policy coordination and 
advisory role that functions across all levels of the federal government and 
answers directly to the President. 

In 2006, the HSC established the National Exercise Program (NEP), and 
approved its charter, implementation plan, and 5 -year national exercise 
schedule in January 2007. The NEP requires that HSEEP methodology 
and guidelines be incorporated into TOPOFF exercise design, evaluation, 
and documentation activities. The NEP charter addresses White House 
recommendations contained in the Katrina Report, a comprehensive 
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assessment of the response and recovery efforts as a result of Hurricane 
Katrina, and provides a framework for federal interagency exercise 
coordination. 

FEMA's National Exercise Division (NED) is the executive agent for the 
NEP. The NEP categorizes exercise activities into four tiers. These tiers 
reflect the relative priority for interagency participation, with Tier I as the 
most important and Tier IV being the least: 

• Tier I: White House-directed exercises with U.S. Government- 
wide strategy, policy focus, and participation; 

• Tier II: U.S. Government- wide priorities, federal strategies and 
policy focus, and significant exercise simulation; 

• Tier III: other federal exercises that have an operational, tactical, or 
organizational focus, and include simulation; and, 

• Tier IV: state, territorial, local, tribal, or private sector exercises. 

The NEP chairs the Executive Steering Committee (ESC), which defines 
the exercise schedule, supports coordination of exercises, and ensures that 
exercises are conducted. The ESC coordinates departmental and agency 
roles and responsibilities, and reports to the HSC's Exercise and 
Evaluation Sub-Interagency Policy Committee (Sub-IPC). The Sub-IPC 
recommends exercise priorities, goals, objectives, schedules, and 
corrective action issues to the HSC Deputies Committee, which is 
composed of deputy cabinet secretaries of the relevant federal agencies. 
The ESC comprises eight departments and agencies: 

• Department of Defense 

o Office of the Secretary of Defense/Homeland Defense 
o Joint Staff- J7 

• Department of Homeland Security 

• Department of Energy 

• Department of Health and Human Services 

• Department of Justice 

o Federal Bureau of Investigation 

• Department of Transportation 

• Office of the Director of National Intelligence 

• Federal Bureau of Investigation 

The ESC may also contain up to two rotating members who serve 1-year 
terms. In addition to their program responsibilities, the NED and NEP are 
responsible for the following: 

• HSEEP, which develops policy and guidance on all aspects of 
preparedness exercises; 
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• The National Exercise Schedule, which encompasses federal, state, 
territorial, local, tribal, or private sector Tier I through Tier IV 
exercises; 

• The CAP system, which provides an interdepartmental system to 
develop, prioritize, and track corrective actions following exercises 
and real-world events; 

• The Lessons Learned Information Sharing (LLIS) System, a web- 
based national network of lessons learned and best practices for the 
emergency response and homeland security community; 

• The Regional Exercise Support Program, which supports 
regionally coordinated exercises through assistance in the design, 
development, conduct, and evaluation of preparedness exercises; 

• The Terrorism Prevention Exercise Program, which conducts 
exercises and supports activities that increase awareness, 
cooperation, and information sharing among homeland security 
and law enforcement officials at all levels of government; and 

• The Senior Official Exercise, which supports the design, 
development, conduct, and evaluation of principal level and intra- 
DHS exercises. 



Results of Review 

TOPOFF Exercises Did Not Include a Requirement for Corrective 
Action Planning Prior to 2007 

Past TOPOFF exercises and actual disasters such as Hurricane Katrina have 
uncovered gaps in the Nation's preparedness. Federal, state, territorial, and local 
response organizations have recognized the importance of promoting best 
practices and identifying areas for improvement to produce a more effective 
response to acts and threats, as well as natural disasters. However, a process to 
prioritize, track, and validate corrective actions taken to mitigate weaknesses 
identified in exercise planning, development, and execution was not established 
until 2007. 

Prior to 2007, HSEEP's exercise planning, evaluation, after-action 
implementation planning, and a corrective action process were incomplete. 
During TOPOFF 2 in May 2003, attendees and exercise participants identified 
several lessons learned relating to exercise design and conduct. However, 
exercise evaluation and improvement planning was not performed, as these 
actions were not included as part of the initial HSEEP guidance. 

TOPOFF 3, in 2005, was the first exercise that DHS sponsored with full 
responsibilities for exercise planning, conduct, and after-action activities. The 
exercise results identified operational shortfalls in terrorism prevention, response, 
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and recovery. The TOPOFF 3 after-action process followed 2003 HSEEP 
principles, which focused on determining why a particular event or response 
occurred, and provided participants with information that could be used to 
improve response and recovery efforts. However, improvement planning to 
address remedial needs and corrective action procedures was not a part of 
HSEEP. Therefore, the final AAR recommendations only informed participating 
departments and agencies of existing problems, and encouraged improvements in 
agency prevention, response, and recovery capabilities. 

In the absence of a process to identify and implement corrective actions, problem 
areas identified during and even prior to the TOPOFF 3 exercise recurred during the 
response to Hurricane Katrina in 2005, and again in the 2007 TOPOFF 4 exercise. 

A Corrective Action Program Has Not Been Fully Implemented 

Previous preparedness exercises and real-world incidents did not result in the 
development of corrective action plans or the assignment of responsibility for 
corrective actions to resolve cross-cutting issues, because no department or 
agency had authority to direct the actions of another. Participating agencies 
developed internal corrective action plans, but without a coordinated interagency 
process, issues involving multiple federal entities remained unresolved. 

In February 2007, HSEEP updated the evaluation and improvement planning 
section to include (1) more details on improvement planning, and (2) the 
requirement for a corrective action planning system. In addition, the NEP 
requires the development of a corrective action process to ensure that lessons 
from exercises are either sustained or improved as appropriate. 

The CAP provides the basis for systematically developing, prioritizing, and 
tracking corrective actions following exercises, real-world events, and policy 
discussions. Established as part of the NEP Implementation Plan, the CAP 
represents an improvement toward interagency coordination and accountability 
for corrective action after TOPOFF and other exercises. It requires that all 
departments and agencies maintain a corrective action program that can generate 
data for, and track assignments from, the NEP CAP for unclassified issues. 
Appendix C provides a diagram of the CAP process. 

The full implementation of a CAP system involves the coordination of activities 
between the HSC, ESC, NEP, and participating departments and agencies. The 
HSC maintains a policy coordination and advisory role that functions across all 
levels of the federal government, and answers directly to the President. The HSC 
also maintains responsibility for corrective action issues and communicating the 
overall direction of CAP to department and agency top officials. The ESC 
coordinates departmental and agency roles and responsibilities, and reports to the 
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HSC's Exercise and Evaluation Sub-IPC. The NEP is responsible for 
incorporating HSEEP guidelines into TOPOFF exercise design and evaluation. 

The TOPOFF 4 AAR, which has not been approved or released by DHS, concludes 
that NEP has established a process for tracking and monitoring the implementation 
of corrective actions. In June 2008, the ESC met to prioritize corrective actions 
identified in the March 2008 AAR Implementation Plan. An NEP official informed 
that in September 2008, the Sub-IPC reviewed the corrective actions identified in 
the AAR Implementation Plan and transmitted them to the HSC Domestic 
Readiness Group for approval. Subsequently, the HSC-approved corrective actions 
were assigned to individual departments and agencies for remediation. The official 
added that NED entered the corrective actions into the CAP system and initiated 
training to ensure that action officers can enter information on the status of assigned 
tasks. However, as of March 2009, the CAP system does not include corrective 
action updates and has not produced required progress reports to validate 
completion of any corrective actions. 

In an August 2007 memorandum to cabinet and agency heads, the Secretary of 
DHS emphasized the NEP Implementation Plan requirement for adherence to a 
disciplined CAP that outlines the process by which shortfalls are identified, tasked 
for action, and tracked to resolution. NEP managers said that generating full 
support and participation for the CAP system from department and agency 
officials is a significant challenge. The support and active participation of the 
Sub-IPC and HSC Deputies Committee are also critical to ensuring that corrective 
actions are assigned, tracked, and validated. 



Recommendations 

We recommend that the Administrator, Federal Emergency Management 
Agency: 

Recommendation #1 : Provide the interagency Policy Coordinating 
Committees and the HSC with detailed information regarding challenges 
faced and actions that need to occur for complete implementation of the 
CAP system. 

Recommendation #2 : Request that the Sub-IPC and the HSC use the 

information provided to clarify and communicate to federal agencies and 
departments their responsibilities for corrective action assignments, 
tracking, and validation. 

Recommendation #3 : Revise the current NEP Implementation Plan by 
including specific program control objectives and procedures that will 
provide management reasonable assurance on CAP implementation. 
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Management Comments and OIG Analysis 



FEMA Response to Recommendation #1: FEMA agreed with our 
recommendation and will provide a detailed corrective action plan for 
implementation in their 90-day letter following the issuance of our final 
report. 

OIG Analysis: This recommendation is resolved and open pending our 
receipt of FEMA' s 90-day letter. 

FEMA Response to Recommendation #2: FEMA agreed with our 
recommendation and will provide a detailed corrective action plan for 
implementation in their 90-day letter following the issuance of our final 
report. 

OIG Analysis: This recommendation is resolved and open pending our 
receipt of FEMA' s 90-day letter. 

FEMA Response to Recommendation #3: FEMA agreed with our 
recommendation and will provide a detailed corrective action plan for 
implementation in their 90-day letter following the issuance of our final 
report. 

OIG Analysis: This recommendation is resolved and open pending our 
receipt of FEMA' s 90-day letter. 



Recurring Themes Identified in Previous Exercises and Real- World 
Events Have Not Been Resolved 

According to the April 2007, National Exercise Program Implementation Plan, 

"The perennial lament of exercise planners and participants is 
exercise fatigue, when multiple exercises require recurring 
application of a department or agency's limited time and 
resources. This fatigue factor is exacerbated by frustrations 
that occur when exercise activities do not lead to appreciable 
improvements in policies, plans, or performance. " 

The TOPOFF 4 AAR concluded that the exercise identified (1) where the 
Nation's preparedness has improved, and (2) areas that need further improvement. 
The areas that need further improvement include recurring themes that have been 
identified in previous exercises and real- world events, and several new areas. 
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The Homeland Security Advisory System (HSAS) provides an example of a 
policy issue repeatedly identified during TOPOFF exercises that has not been 
resolved. The HSAS was designed to guide protective measures when specific 
threat information from a particular sector or geographic region is received. It 
combines threat information with vulnerability assessments, and provides 
pertinent information to public safety officials and the public. Following 
TOPOFF 2, the AAR reported uncertainty regarding protective actions to be taken 
for each HSAS threat level. Specifically, the purpose, definitions, and 
consequences of the HSAS threat levels were not clear. In addition, coordination 
of actions taken by other agencies or jurisdictions under HSAS was also unclear. 
Participants at all levels of government, as well as international participants, have 
raised questions about the meaning and implications of HSAS level changes. 

In the aftermath of TOPOFF 3, the AAR concluded that the HSAS system was 
still not understood or systematically used by exercise participants. However, no 
corrective actions were included in the AAR, since this was not required by the 
exercise evaluation and improvement process. After TOPOFF 4, the AAR 
concluded again that the purpose, definitions, and consequences of HSAS threat 
levels remain unclear. 

Regarding the reoccurrence of HSAS issues in successive TOPOFF exercises, 
NEP program managers said that actions to address HSAS issues required 
correcting unresolved policy issues beyond the scope and authority of their office. 
Following TOPOFF 4, NEP officials reported that the HSC had prioritized HSAS 
issues for immediate attention. However, a decision has not been made whether 
actions to address HSAS would be included and tracked under the CAP, or 
managed and monitored directly by the HSC. The HSC, through its Domestic 
Readiness Group and Sub-IPC, can prioritize and assign action under CAP. 



Recommendations 

We recommend that the Administrator, Federal Emergency Management 
Agency: 

Recommendation #4 : Request that the HSC clarify and document their 
determination whether issues prioritized by the Sub-IPC for immediate 
action after TOPOFF exercises will be included and tracked under the CAP. 
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Management Comments and OIG Analysis 



FEMA Response to Recommendation #4: FEMA agreed with our 
recommendation and will provide a detailed corrective action plan for 
implementation in their 90-day letter following the issuance of our final 
report. 

OIG Analysis: This recommendation is resolved and open pending our 
receipt of FEMA' s 90-day letter. 

Participation and Support in TOPOFF Planning and Implementation 
Needs To Be Increased 

The participation of top officials responsible for response and consequence 
management is critical to the success of TOPOFF exercises. Based on the April 
2007 NEP Implementation Plan, all departments and agencies must budget for 
support to NEP Tier I exercises and be prepared to provide Principal- or Deputy- 
level support. However, FEMA officials said that not all federal departments and 
agencies have (1) established their participation on committees and working 
groups, (2) accounted for the costs associated with participation, and 
(3) committed sufficient resources for TOPOFF exercises. 

For private sector participation, the TOPOFF 4 AAR concludes that exercises 
offer the private sector an opportunity to collaborate with the government and test 
plans for disaster response and recovery. However, planners said that to create 
strong linkages, the private sector needed to be engaged earlier in the planning 
cycle with their federal agency counterparts. 

Federal Participation 

Recognizing that top officials rarely participated in national-level 
preparedness exercises, Congress required that top officials and key staff 
participate fully in exercise planning, implementation, and after-action 
exercise activities. At the federal level, NEP engaged more than 40 
departments, agencies, and offices in its TOPOFF 4 interagency planning 
process. However, all agencies did not commit the required personnel or 
resources to accomplish TOPOFF exercise activities, which affected the 
extent of their participation in the planning, implementing, and evaluating 
processes. 

In August 2007, 10 weeks prior to the start of TOPOFF 4, Secretary 
Chertoff distributed a memorandum for cabinet and agency heads 
requesting those who had not assigned an executive to the ESC to do so. 
However, by the conclusion of the TOPOFF 4 exercise in October 2007, 
not all federal departments and agencies had designated representatives to 
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the ESC or made budgetary decisions to ensure the availability of 
resources to plan and participate in the exercise. NEP officials said that at 
the conclusion of TOPOFF 4, departmental and agency participation was 
not at the level expected for robust interagency planning and participation. 
As a result, many of the TOPOFF 4 planning and scenario design 
activities did not have a broad representation of exercise planners. 

Although TOPOFF objectives are expected to reflect U.S. Government- 
wide priorities, some exercise objects were not emphasized because of the 
need for more inclusive participation in exercise plan, design, or after- 
action analysis. In the exercise after-action discussions, ESC members 
said that the need for information and participation by departments and 
agencies has created a perception that the TOPOFF exercise program is 
overly focused on DHS objectives and is of minimal value. 

NEP managers described securing sufficient federal support for TOPOFF 
as a primary challenge to establishing a national exercise program. The 
TOPOFF 4 AAR suggests that HSEEP policy and guidance be amended to 
emphasize the requirements for TOPOFF participation at the highest 
levels. Achieving early and active engagement of all concerned parties 
will help ensure that TOPOFF exercises reflect federal government-wide, 
state, territorial, local, and private sector priorities. FEMA will need the 
support and assistance of federal interagency committees and the HSC to 
achieve this goal. 

Participation at the State, Territorial, Local, and Private Sector 
Levels 

State, territorial, and local officials responsible for TOPOFF 4 planning 
and implementation commented that they: 

• Were not engaged early enough in the exercise planning process; 

• Were not given clear benchmarks and guidelines for participation; 
and 

• Would benefit from more support and mentoring from state-level 
counterparts from previous TOPOFF exercises, which was a part 
of the TOPOFF 3 preparation process. 

NED supported and managed the needs of the private sector working 
group for the TOPOFF 3 exercise. This responsibility was not clearly 
delineated for TOPOFF 4. According to a FEMA official, the NEP did 
not plan or budget for the necessary recruitment and engagement efforts of 
the private sector in TOPOFF 4. 

TOPOFF 4 private sector participants included a broad range of 
representatives from the critical infrastructure/key resources sector, large 
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corporations, nongovernmental and service organizations, and others. 
Their participation in TOPOFF helped to identify areas where they could 
contribute during or after an actual terrorism event or a major natural 
disaster, and how national-level policies could be integrated into their 
operations. 

Following TOPOFF 4, private sector participants provided the following 
assessments: 

• Exercise planners needed to distinguish the levels of participation 
of each type of private sector organization and clarify the 
objectives and requirements of participation for each 
organizational type; and 

• Private sector participants needed to be recruited earlier in the 
exercise design and plan cycle. 

During TOPOFF 4, there was confusion within the private sector about 
specific roles and defined expectations during the exercise. In the absence 
of a clear delineation of duties and specific roles, private sector entities 
developed expectations that did not coincide with those of the federal 
government. 

Recommendations 

We recommend that the Administrator, Federal Emergency Management 
Agency: 

Recommendation #5 : Task FEMA's Private Sector Office to co-chair a 
working group with DHS private sector entities to identify the requirements 
for private sector TOPOFF participation, and establish a plan to support 
their participation. 

Recommendation #6 : Develop and implement a plan to secure the full 
participation and required resources of federal departments and agencies, 
and state, local, and territorial entities to design, plan, and conduct the 
TOPOFF exercise program. 

Recommendation #7 : Amend National Exercise Program guidance to 
ensure that it addresses and supports the appropriate level of federal 
departments and agencies, and state, local, and territorial participation 
expected in the TOPOFF exercise program. 
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Management Comments and OIG Analysis 



FEMA Response to Recommendation #5: FEMA agreed with our 
recommendation, and will provide a detailed corrective action plan for 
implementation in their 90-day letter following the issuance of our final 
report. 

OIG Analysis: This recommendation is resolved and open pending our 
receipt of FEMA' s 90-day letter. 

FEMA Response to Recommendation #6: FEMA agreed with our 
recommendation, and will provide a detailed corrective action plan for 
implementation in their 90-day letter following the issuance of our final 
report. 

OIG Analysis: This recommendation is resolved and open pending our 
receipt of FEMA' s 90-day letter. 

FEMA Response to Recommendation #7: FEMA agreed with our 
recommendation, and will provide a detailed corrective action plan for 
implementation in their 90-day letter following the issuance of our final 
report. 

OIG Analysis: This recommendation is resolved and open pending our 
receipt of FEMA' s 90-day letter. 

Evaluator Selection and Training Needs Improvements 

The TOPOFF 4 Evaluator Handbook outlines evaluator responsibilities that 
include recording observations of player actions and managing data collection at 
all exercise sites. Data collected by evaluators during the TOPOFF 4 exercise 
supported the post-exercise reconstruction and analysis process, the development 
of observations, lessons learned, and the recommendations contained in the 
TOPOFF 4 AAR. More than 650 evaluators participated in the TOPOFF 4 
exercise. 

The TOPOFF 4 Evaluator Handbook requires evaluators to be familiar with the 
roles and responsibilities of the players they observe. NED relied on departments 
and agencies to identify and assign individuals with appropriate agency-specific 
knowledge to perform data collection at key locations during the exercise. 
However, some departments, agencies, and DHS components did not assign 
evaluators to support the evaluation methodology until just before the full-scale 
exercise. Therefore, all evaluators did not complete the NED evaluator training, 
or familiarize themselves with the exercise plan, scenario, missions, policies, and 
procedures specific to their assigned location. 
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TOPOFF 4 evaluators were offered a training session conducted by NED in 
advance of the exercise. Agency personnel selected as evaluators were expected to: 

• Review the exercise plan and scenario, the master scenario events list, 
evaluation plan, and control staff instructions; 

• Complete evaluator training requirements; 

• Familiarize themselves with the missions, plans, policies, procedures, and 
processes applicable to their assigned location; 

• Familiarize themselves with data collection forms and exercise evaluation 
guides for their assigned location; and 

• Attend the controller/evaluator briefings at their assigned location. 

Some of the evaluators said that they were selected by their departments and 
agencies to serve as evaluators shortly before the start of the TOPOFF 4 exercise. 
Therefore, they were unable to complete the full training program conducted by 
NED. They participated in an abbreviated training session offered the day before 
the exercise started, and did not have the opportunity to review the master 
scenario events list, evaluation plan, or control staff instructions before assuming 
their evaluator duties. This resulted in reduced assurance that evaluators were 
able to obtain the necessary knowledge, skills, and abilities to record required data 
during the exercise. 

Evaluators without sufficient knowledge of the exercise plan, scenario, agency 
plans, policies, and procedures were not as effective in performing their assigned 
duties during the exercise. For example, evaluators collecting data at the Portland 
Joint Information Center and the Joint Federal Office said that they did not have a 
good grasp of the evaluation objectives for their assigned locations, but were 
making their best effort to collect data and record what they observed. 

The TOPOFF 4 AAR Annex 1 includes comments and recommendations for 
improvements from the exercise working groups. The working groups 
acknowledged in their comments that the complexity and specific requirements of 
TOPOFF created a need for more extensive evaluator training tailored to the 
requirements of each exercise venue. They also proposed that evaluators be 
identified earlier in the exercise planning process, and that the training schedule 
be integrated into venue and interagency group meetings. 

Recommendations 

We recommend that the Administrator, Federal Emergency Management 
Agency: 

Recommendation #8 : In response to TOPOFF 4 AAR Annex 1 
recommendations, develop and implement a plan to identify, train, and 
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support department and agency officials and staff who will control, 
evaluate, or participate in TOPOFF exercise programs. 



Management Comments and OIG Analysis 

FEMA Response to Recommendation #8: FEMA agreed with our 
recommendation, and will provide a detailed corrective action plan for 
implementation in their 90-day letter following the issuance of our final 
report. 

OIG Analysis: This recommendation is resolved and open pending our 
receipt of FEMA' s 90-day letter. 



Dissemination of TOPOFF AARs, Lessons Learned, and Best Practices 

According to HSEEP guidelines, the overarching TOPOFF exercise objectives are 
(1) the identification and dissemination of lessons learned, and (2) the promotion 
of best practices to the national community of first responders and emergency 
managers. In addition, HSPD-8: Terrorism Preparedness mandates that DHS 
"develop and maintain a system to collect, analyze, and disseminate lessons 
learned, best practices, and information from exercises, training events, research, 
and other sources." DHS has not yet developed an effective process to 
disseminate TOPOFF AARs, lessons learned, and best practices to a broad 
national audience. 

Distribution of TOPOFF AARs 

For the TOPOFF 4 exercises ending in November 2007, NEP finalized 
and submitted the AAR to the HSC Sub-IPC for review in March 2008. In 
June 2008, the report was transmitted to FEMA officials for a security 
review and classification. In November 2008, FEMA transmitted the 
AAR to DHS headquarters for final approval and distribution. However, 
as of March 2009, DHS had not approved the TOPOFF 4 AAR for release 
to participating agencies and other interested parties. 

An NEP official said that the current HSEEP timeline requires an AAR to 
be produced 60 days after the exercise. Based on feedback received from 
federal, state, and local officials, the 60-day requirement will be changed 
to 90 days. Also, taking into account the additional interagency 
coordination, the NEP official recommended the following timeframe: 

(1) draft interagency report is created within 120 days after the exercise; 

(2) FEMA approves the report within 150 days after the exercise; and 

(3) DHS approves the report within 180 days after the exercise. 
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To date, TOPOFF AARs have had restricted distribution to a limited 
number of federal, state, territorial, and local departments and agencies on 
a loan basis because DHS considers that the reports may contain sensitive 
information. All TOPOFF AARs have been designated For Official Use 
Only (FOUO), which precludes their dissemination to a broad national 
audience. 

Distribution of TOPOFF Lessons Learned 

The Lessons Learned Information Sharing (LLIS) website is DHS' online 
repository for information dedicated to the emergency response and 
homeland security communities. NEP managers describe the LLIS 
website as the primary method to disseminate TOPOFF exercise lessons 
learned and best practices. 

The TOPOFF 4 exercise plan did not encompass specific activities or a 
budget to disseminate lessons learned immediately following the exercise. 
However, to bring the results of the exercise to a wider national audience, 
senior FEMA officials directed the NED to conduct a 1-day National 
After-Action Conference for state homeland security advisors and 
emergency management directors. NED invited officials from 56 states 
and territories to attend the March 2007 event in Oklahoma City, 
Oklahoma. The conference goals were to report on the TOPOFF 4 
exercise, present lessons learned, and describe the new CAP process. The 
TOPOFF 4 National After- Action Conference attracted 142 participants 
from federal, state, and territorial agencies; 47 attended in person and 95 
participated via a webcast of the conference. Conference participants 
represented 26 states and territories. 

The conference included a 30-minute presentation and a FOUO document 
that described lessons learned from the TOPOFF 4 exercise. NED 
officials said that while participation was less than envisioned, the 
conference achieved its goals, and was an effective way to disseminate 
TOPOFF lessons learned to a national audience of state homeland security 
officials. 

According to the GAO Standards for Internal Control in the Federal 
Government, information should be communicated to management and 
others who need it within a timeframe that enables them to carry out their 
responsibilities. NED managers said that the process to disseminate 
TOPOFF AARs and lessons learned is evolving, and progress will be 
made as HSEEP-directed exercise evaluation and improvement planning 
becomes more standardized under the NEP and CAP. 
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Recommendations 



We recommend that the Administrator, Federal Emergency Management 
Agency: 

Recommendation #9 : With assistance from DHS senior officials and 
offices, develop and implement a process and timeframe to distribute final 
TOPOFF AARs. 

Recommendation #10 : To communicate information to a national 
audience of first responders and emergency management officials, develop 
and implement a process and timeframe to populate LLIS.gov with 
TOPOFF exercise lessons learned and best practices. 



Management Comments and OIG Analysis 

FEMA Response to Recommendation #9: FEMA agreed with our 
recommendation, and will provide a detailed corrective action plan for 
implementation in their 90-day letter following the issuance of our final 
report. 

OIG Analysis: This recommendation is resolved and open pending our 
receipt of FEMA' s 90-day letter. 

FEMA Response to Recommendation #10: FEMA agreed with our 
recommendation, and will provide a detailed corrective action plan for 
implementation in their 90-day letter following the issuance of our final 
report. FEMA noted in their response memorandum that at the conclusion 
of TOPOFF 4, DHS/FEMA produced and disseminated three Quick Look 
reports for participating departments and agencies and the response 
community, as well as a public version. 

OIG Analysis: This recommendation is resolved and open pending our 
receipt of FEMA' s 90-day letter. 
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Appendix A 

Purpose, Scope, and Methodology 



Our objective was to determine whether the DHS is achieving its goals to 
identify, distribute, and address lessons learned and corrective needs that 
surfaced during the biennial national TOPOFF terrorism preparedness 
exercises. We examined these processes in relation to the TOPOFF 3 and 
TOPOFF 4 exercises. 

We conducted interviews with DHS/FEMA National Preparedness 
Directorate staff who manage the TOPOFF exercise series. We interviewed 
other DHS officials, managers, and staff; personnel from other federal 
departments and agencies; and state, local, and private sector exercise 
participants who played an instrumental role in TOPOFF exercise design, 
planning, and evaluation. We collected, reviewed, and analyzed documents 
related to exercise planning, training, evaluation, after-action process, and 
the corrective action program. 

We observed and collected data during the TOPOFF 4 exercise in 
Phoenix, Arizona, and Portland, Oregon; and at the exercise master 
control cell established in Northern Virginia. We observed the TOPOFF 4 
exercise after-action briefings (hot washes) conducted in Phoenix and 
Portland, and in Virginia; the December 2007 Long-term Recovery Table 
Top Exercise; the January 2008 After- Action Conference; and the April 
2008 National After-Action Conference in Oklahoma City, Oklahoma. 

We conducted our review between October 2007 and August 2008 under 
the authority of the Inspector General Act of 1978, as amended, and 
according to the Quality Standards for Inspections issued by the 
President's Council on Integrity and Efficiency. 
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Appendix B 

Management Response to the Draft Report 



U.S. Department of Homeland Security 
Washington. DC 20472 




I FEMA 



March 20, 20051 



MEMORANDUM FOR: 



Richard L. Skinner 
Inspector General 



FROM: 



Robert A. Farmer ^JU^\. 
Acting Director 

Office of Policy & Program Analysis 




draft audit report. We have provided technical comments under separate cover. As the Federal Emergency 
Management Agency (FEMA) works toward refining its programs, the OIG's independent analysis of 
program performance greatly benefits our ability to continuously improve our activities. 

FEMA concurs with the draft report's ten recommendations. However, regarding the recommendation that 
FEMA develop and implement a process and timeframe to populate LLIS.gov with TOPOFF exercise lessons 
learned and best practices, we note that at the conclusion of TOPOFF 4, DHS/FEMA produced and 
disseminated three Quick Look reports, Participating Departments and Agencies, Response Community, and 
a Public Version. We will continue to enhance this process through the maturation of the National Exercise 
Program, In our 90-day letter following the issuance of your final report, we will provide a detailed corrective 
action plan for implementation of each of the report's ten recommendations. 

We also wish to point out that while the draft report mentions TOPOFF exercises from 2000 until 2007, the 
TOPOFF program was originally housed within the U.S. Department of Justice, then the Department of 
Homeland Security. FEMA did not inherit responsibility for TOPOFF until 2006, with the passage of the 
Post-Katrina Emergency Management Reform Act of 2006. 

The draft report also states that best practices and lessons learned as a result of TOPOFF exercises have not 
been disseminated. We have provided detail on how these lessons learned have been disseminated on 
LLJS.gov to include Quick Look Reports and exercise summaries for all four TOfOFF exercises. 

Finally, the draft report states that the TOPOFF 4- exercise plan did not encompass specific activities or a 
budget to disseminate lessons learned immediately following the exercise. We note however, that the 
TOPOFF 4 Control and Evaluation Working Group had a comprehensive After Action Report process which 
included the development and dissemination of three Quick Look Reports. 



DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 



www.fcma.sov 



Page 20 



Appendix B 

Management Response to the Draft Report 



Thank you again for the opportunity to comment on this draft report and we look forward to working with 
you on other issues as we both strive to make FEMA the nation's Preeminent Emergency Management and 
Preparedness Agency. 
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Appendix C 

The Corrective Action Program Process 



Step 1 



Event Evaluation 



Evaluafore observe and 
analyze exercise, and 
develop preliminary 
recommendations to 
address observed issues 




Step 2 



After Action 
Conference 



Exercise Participants 

review DRAFT MR and 
• develop proposed CAs to 
address preliminary 
recommendations 



E+30 



Final AAPJIP with specific 
CAs, action officers, start 
dates and completion dates 

r" 



Step 3 



Step 4 



Issue Prioritization H Issue Assignment 



NEPESC prioritizes CAs 
and sends to DRG for 
finalization; DRG sends 
CAs to affected PCC for 
formal assignment 



! 

iE+60 

i 

J 



—I List of prioritized CAs — 



Affected PCC formally 
assigns priority CAs to 
appropriate D/A via 
ExecSec channels 



T 



lE+90 

i 

l 

T 



Formal notification of CA 
assignments to 
appropriate D/As 



Step 5 



Solution 
Development 



Assigned DM develops 
■ Implementation Plan for 
assigned CA 



E+130 



Implementation Plan 
with defined timelines 
and milestones 



Step 6 



Solution Review 



Affected PCC reviews 
implementation plan and 
provides feedback 



T 



E+160 



Final Implementation 
Plan reflecting PCC 
feedback 



Step 7 



Solution 
Implementation 



Assigned D/A 

Implements CAs In 
accordance with proposed 
timelines and milestones 



T 



E+170 



Progress toward CA 
implementation tracked in 
CAP System 

f 



Step? 



Solution 
Validation 



Assigned D/A validates 
implemented CA 
through metrics defined 
in implementation plan 



T 



Ongoing 



E.g., Exercise 
objectives testing 
and validating CAs 



Step 9 



Solution 
Reporting 



NEPESC complies and 
distributes repots to 
D/As, PCCs, and HSC 
Deputies Committee 

— r 



i 

I Monthly (D/As) 

Quarterly (PCCs) 
| Biannually (Deputies) 



CA implementation 
summarized and 



Key Acronyms 



After-Action Report (AAR) 
Improvement Plan (IP) 
Corrective Action (CA) 
Department/Agency (D/A) 
Homeland Security Council (HSC) 
Executive Secretary (Exec Sec) 
DRG (Domestic Readiness Group) 
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Appendix D 

Major Contributors to This Report 



Deborah Outten-Mills, Chief Inspector, Department of Homeland 
Security, Office of Inspections 

Jim O'Keefe, Senior Inspector, Department of Homeland Security, Office 
of Inspections 

Lawrence Anderson, Inspector, Department of Homeland Security, Office 
of Inspections 

Jessica Barnes, Inspector, Department of Homeland Security, Office of 
Inspections 
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Appendix E 
Report Distribution 



Department of Homeland Security 

Secretary 

Acting Deputy Secretary 

Chief of Staff for Operations 

Chief of Staff for Policy 

Acting General Counsel 

Executive Secretariat 

Director, GAO/OIG Liaison Office 

Assistant Secretary for Policy 

Assistant Secretary for Public Affairs 

Assistant Secretary for Legislative Affairs 

Administrator, Federal Emergency Management Administration 

Audit Liaison, Federal Emergency Management Administration 

Office of Management and Budget 

Chief, Homeland Security Branch 
DHS OIG Budget Examiner 

Congress 

Congressional Oversight and Appropriations Committees, as appropriate 
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ADDITIONAL INFORMATION AND COPIES 

To obtain additional copies of this report, please call the Office of Inspector General (OIG) at (202) 254-4199, 
fax your request to (202) 254-4305, or visit the OIG web site at www.dhs.gov/oig. 



OIG HOTLINE 

To report alleged fraud, waste, abuse or mismanagement, or any other kind of criminal or noncriminal 
misconduct relative to department programs or operations: 

• Call our Hotline at 1-800-323-8603; 

• Fax the complaint directly to us at (202) 254-4292; 

• Email us at DHSOIGHOTLINE@dhs.gov; or 

• Write to us at: 

DHS Office of Inspector General/MAIL STOP 2600, 
Attention: Office of Investigations - Hotline, 
245 Murray Drive, SW, Building 410, 
Washington, DC 20528. 



The OIG seeks to protect the identity of each writer and caller. 



